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(54) Abstract Title 

Tracking speakers in an audio stream 



(57) Audio information is processed to identify potential segment boundaries, corresponding to a speaker 
changes 220. Thereafter, homogeneous segments (generally corresponding to the same speaker) are clustered 
230, and a cluster identifier is assigned to each identified segment. A segmentation subroutine identifies 
potential segment boundaries using the BIC model selection criterion. A window selection scheme considers a 
relatively small amount of data in areas where new boundaries are very likely to occur, and the window size is 
increased when boundaries are not very likely to occur. When a segment boundary is found in a window, the 
next window begins after the detected boundary, using the minimal window size. BIC tests can be eliminated 
when they correspond to locations where the detection of a boundary is very unlikely. 
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METHODS AND APPARATUS FOR TRACKING SPEAKERS 
IN AN AUDIO STREAM 

The present invention relates generally to audio information classification 
systems and, more particularly, to methods and apparatus for identifying 
speakers in an audio file. 

Many organizations, such as broadcast news organizations and information 
retrieval services, must process large amounts of audio information, for 
storage and retrieval purposes. Frequently, the audio information must be 
classified by subject or speaker name, or both. In order to classify audio 
information by subject, a speech recognition system initially transcribes 
the audio information into text for automated classification or indexing. 
Thereafter, the index can be used to perform query-document matching to 
return relevant documents to the user. 

Thus, the process of classifying audio information by subject has 
essentially become fully automated. The process of classifying audio 
information by speaker, however, often remains a labor intensive task, 
especially for real-time applications, such as broadcast news. While a 
number of computationally-intensive off-line techniques have been proposed 
for automatically identifying a speaker from an audio source using speaker 
enrollment information, the speaker classification process is most often 
performed by a human operator who identifies each speaker change, and 
provides a corresponding speaker identification. 

The segmentation of audio files is also useful as a preprocessing step for 
a speaker identification tool that actually provides a speaker name for 
each identified segment. In addition, the segmentation of audio files may 
be used as a preprocessing step to reduce background noise or music. 

As apparent from the above-described deficiencies with conventional 
techniques for classifying an audio source by speaker, a need exists for a 
method and apparatus that automatically classifies speakers in real-time 
from an audio source. A further need exists for a method and apparatus 
that provides improved speaker segmentation and clustering based on the 
Bayesian Information Criterion (BIC) . 

The present invention accordingly provides, in a first aspect, a method for 
tracking a speaker in an audio source, said method comprising the steps of: 
identifying potential segment boundaries in said audio source; and 
clustering homogeneous segments from said audio source substantially 
concurrently with said identifying step. 



4B/C- = - § bg\l J + j logll/l + !L J- L N^.l 

where IU is the determinant of the covariance of the window of all n 
samples, 11,1 is the determinant of the covariance of the first suhdrvrsron 
o£ ehe window, and |I S I is the determinant of the covariance of the second 
subdivision of the window. 

Preferably, said identifying step considers a smaller window size, n. of 
samples in areas where a segment boundary is unlikely to occur. 

Preferably, said window size, n, is increased in a relatively slow manner 
Preferably, increa ses in a faster manner when the 

when the window size is small and increases 

window size is larger. 

ie init-ialized to a minimum value after a 
Preferably, said window size, n, is initialized 

segment boundary is detected. 

Pre£ eraol y . said BIC model section test Is not perked at the herder of 
each window of samples. 

Preferably, said BIC model selection test is not performed when the window 
sire, n, exceeds a predefined threshold. 

The method of claim 1. wherein sard clusterin, step rs performed usin 3 a 
BIC model-selection criterion. 

Preferably, a first model assumes that fwc serenes or clusters shouid be 
ZU and a second model assumes that sard two saints or clusters 
should be maintained independently. 



A method according to the first aspect preferably further comprises the 
step of merging said two clusters if a difference in BIC values for each of 
said models is positive. 

Preferably, said clustering step is performed using K previously identified 
clusters and M segments to be clustered. 

A method according to the first aspect preferably further comprises the 
step of assigning a cluster identifier to each of said clusters. 

A method according to the first aspect preferably further comprises 
the step of processing said audio source with a speaker identification 
engine to assign a speaker name to each of said clusters. 

In a second aspect, the present invention provides a method for tracking a 
speaker in an audio source, said method comprising the steps of: 
identifying potential segment boundaries in said audio source; and 
clustering segments from said audio source corresponding to the same 
speaker substantially concurrently with said identifying step. 

Preferably, said identifying step identifies segment boundaries using a BIC 
model -select ion criterion. 

Preferably, a first model assumes there is no boundary in a portion of said 
audio source and a second model assumes there is a boundary in said portion 
of said audio source: 

Preferably, said identifying step considers a smaller window size, n, of 
samples in areas where a segment boundary is unlikely to occur. 

Preferably, said BIC model selection test is not performed where the 
detection of a boundary is unlikely to occur. 

Preferably, said clustering step is performed using a BIC model -selection 
criterion, where a first, model assumes that two segments or clusters should 
be merged, and a second model assumes that said two segments or clusters 
should be maintained independently. 

Preferably, said clustering step is performed using K previously identified 
clusters and M segments to be clustered. 

In a third aspect, the present invention provides a method for tracking a 
speaker in an audio source, said method comprising the steps of: 



identifying potential segment boundaries during a pass through said audio 
source; and clustering segments from said audio source corresponding to the 
same speaker during said same pass through said audio source. 

5 Preferably, said identifying step identifies segment boundaries using a BIC 

model-selection criterion. 

Preferably, a first model assumes there is no boundary in a portion of said 
audio source and a second model assumes there is a boundary in said portion 
10 of said audio source. 

Preferably, said identifying step considers a smaller window size, n, of 
samples in areas where a segment boundary is unlikely to occur. 
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Preferably, said BIC model selection test is not performed where the 
detection of a boundary is unlikely to occur. 

Preferably, said clustering step is performed using a BIC model-selection 
criterion, where a first model assumes that two segments or clusters should 
be merged, and a second model assumes that said two segments or clusters 
should be maintained independently. 

Preferably, said clustering step is performed using K previously identified 
clusters and M segments to be clustered. 

In a fourth asoect. the present invention provides a computer program 
comprising program code to, when loaded into a computer and executed, cause 
the computer to perform the steps of a method of any one of the first, 
second and third aspects. 

Generally, a method and apparatus are disclosed for automatically 
identifying speakers from an audio (or video) source. The audio 
information is processed to identify potential segment boundaries, 
corresponding to a speaker change. Thereafter, homogeneous segments 
(generally corresponding to the same speaker) are clustered, and a cluster 
identifier is assigned to each detected segment. Thus, segments 
corresponding to the same speaker should have the same cluster identifier. 
A clustering output file is generated that provides a sequence of segment 
numbers and a corresponding cluster number. A speaker identification 
engine or a human may then optionally assign a speaker name to each 
cluster . 

The present invention concurrently segments an audio file and clusters the 
segments corresponding to the same speaker. A segmentation subroutine rs 
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utilized to identify all possible frames where there is a segment boundary, 
corresponding to a speaker change. A frame represents speech 
characteristics over a given period of time. The segmentation subroutine 
determines whether or not there is a segment boundary at a given frame, i, 
5 using a model selection criterion that compares two models. A first model 

assumes that there is no segment boundary within a window of samples, (x l; 

x n ) , using a single full* covariance Gaussian. A second model assumes 
that there is a segment boundary within a window of samples, (x x , ... xj , 
using two f ull-covar iance Gaussians, with (x x , ... x t ) drawn from the first 
10 Gaussian, and (x itl/ ... x a ) drawn from the second Gaussian. The i" 1 frame is 

a good candidate for a segment boundary if the expression : 



Mid = - %log\L w \ + \ loglZ/l + n 2 1 logll, i 

i if j did + m , 

+ yAl d + 2 J lo S* 



15 is negative, where \Y, W \ is the determinant of the covariance of the whole 

window (i.e., all n frames), JS/I is the determinant of the covariance . of 
the first subdivision of the window, and |Zj is the determinant of the 
covariance of the second subdivision of the window. 

20 According to a further aspect of the invention, a new window selection 

scheme is presented that improves the overall accuracy of segmentation 
processing, especially on small segments. If the selected window contains 
too many vectors, some boundaries are likely to be missed. Likewise, if 
the selected window is too small, lack of information will result in poor 

25 representation of the data. The improved segmentation subroutine of the 

present invention considers a relatively small amount of data in areas 
where new boundaries are very likely to occur, and increases the window 
size when boundaries are not very likely to occur. The window size 
increases in a slow manner when the window is small, and increases in a 

3 0 faster manner when the window gets bigger. When a segment boundary is 

found in a window, the next window begins after the detected boundary, 
using the minimal window size (N 0 ) . 

In addition, the present invention improves the overall processing time by 
35 better selection of the locations where BIC tests are performed. BIC tests 

can be eliminated when they correspond to locations where the detection of 
a boundary is very unlikely. First, the BIC tests are not performed at the 
borders of each window, since they necessarily represent one Gaussian with 
very little data (this apparently small gain is repeated over segment 
40 detections and actually has no negligible performance impact) . In 
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FIG. 4 is a flow chart describing an exemplary clustering subroutine 
performed by the speaker classification system of FIG. 1. 

FIG. 1 illustrates a speaker classification system 100 in accordance with a 
5 preferred embodiment of the present invention that automatically identifies 

speakers from an audio-video source. The audio-video source file may be, 
for example, an audio recording or live feed of a broadcast news program. 
The audio-video source is initially processed to identify all possible 
frames where there is a segment boundary, indicating a speaker change. 

10 Thereafter, homogeneous segments (corresponding to the same speaker) are 

clustered, and a cluster identifier is assigned to each of the identified 
segments. Thus, all of the segments corresponding to the same speaker 
should have the same cluster identifier. The speaker classification system 
100 produces a clustering output file that provides a sequence of segment 

15 numbers (with the start and end times of each segment) along with the 

corresponding identified cluster number. 

A speaker identification engine or a human may then optionally assign a 
speaker name to each cluster. The optional speaker identification engine 
20 uses a pre-enrolled pool of speakers for identification. Since the speaker . 

identification task is an optional component of the speaker classification 
system 100, the present invention does not require training data for each 
of the speakers . 

25 FIG . 1 is a block diagram showing the architecture of an illustrative 

speaker classification system 100 in accordance with a preferred embodiment 
of the present invention. The speaker classification system 100 may be 
embodied as a general purpose computing system, such as the general purpose 
computing system shown in FIG. 1. The speaker classification system 100 

30 includes a processor 110 and related memory, such as a data storage device 

120, which may be distributed or local. The processor 110 may be embodied 
as a single processor, or a number of local or distributed processors 
operating in parallel . The data- storage device 120 and/or a read only 
memory (ROM) are operable to store one or more instructions, which the 

35 processor 110 is operable to retrieve, interpret and execute. 

The data storage device 120 preferably includes an audio corpus database 
150 for storing one or more prerecorded or live audio or video files (or 
both) that can be classified in real-time in accordance with the present 
40 invention. The data storage device 120 also includes one or more cluster 

output files 160, discussed below. In addition, as discussed further below 
in conjunction with FIGS. 2 through 4, the data storage device 120 includes 
a speaker classification process 200, a segmentation subroutine 300 and a 
clustering subroutine 400. The speaker classification process 200 
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h o files in the audio corpus database 150 and 
analyzes one or more audio ^ ing . sequence of segment 

produces d t Les of each se^, along wit, the 

numbers (with the start anu 
corresponding identified cluster number. 

Th e s.ro^e 300 - V ^ZT^Z"~ Mh 

criterion. B^C is an a Y „„ rf „,. b « s t represents II 

.»d to decide which of P parametric models beet 

^ E , ch „odel M, has a nu^er of 

■ B^rina the Dimension of a Mode ^ ^ BIC 

which maximizes 

wc^togLjCxi,....*.)-*^]^ 11 Eq ' (1) 

■m.im likelihood of the data under 
h , re A - 1 and where U is the maximum likelino 

„ the likelihood of the data with maximum 
rao del M, (in other words, the like! ^ ^ ^ _ 

models, a simple test is used positive. 
Mi i. seiected over the „odei M, if ABXC . Bfc - Bfc P _ 

Lital «, the modei Mi is seiectea over the modef M, 



BIC 2 , i-s negative. 



As p — indicated, the speaker ciassificacfo . . ^00 . 
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segment numbers (witn u 

.j ar ,Hfied cluster number, 
the corresponding identified ci 

ficat ion system 100 initially 
As s « in r». ». fh. speaker c - ««« V ^ ^ _ ^ 

extracts the cepstra! features ™ ^^.^ a ^.»t. the data 
information as possible. 



Thereafter, the speaker classification process 200 implements the 
segmentation subroutine 300, discussed further below in conjunction with 
FIG. 3, during step 220 to separate the speakers. Generally, the 
segmentation subroutine 300 attempts to identify all possible frames where 
there is a segment boundary. 

The speaker classification process 200 implements the clustering subroutine 
400, discussed further below in conjunction with FIG. 4, during step 230 to 
cluster the homogeneous segments (corresponding to the same speaker) that 
were identified by the segmentation subroutine 300. Generally, the 
clustering subroutine 400 assigns a cluster identifier to each of the 
detected segments . All of the segments corresponding to the same speaker 
should have the same cluster identifier. 

Finally, the results of the classification system 100 are displayed during 
step 240. Generally, the results are the cluster output file 160 that 
provides a so<juenc& of segment numbers (with the start and end times of 
each segment) along with the corresponding identified cluster number. A 
test is then performed during step 250 to determine if any audio remains to 
be processed. If it is determined during step 250 that some audio does 
remain to be processed, then program control to step 210 and continues 
processing in the manner described above. If, however, it is determined 
during step 250 that there is no audio remaining to be processed, then 
program control terminates during step 260. 

As previously indicated, the speaker classification process 200 implements 
the segmentation subroutine 3 00 (FIG. 3) during step 22 0 to identify all 
possible frames where there is a segment boundary. Without loss of 
generality, consider a window of consecutive data samples (x i# . . . x n ) in 
which there is at most one segment boundary. 

The basic Question of whether or not there is a segment boundary at frame i 
can be cast as a model selection problem between the following two models: 
model Mi, where (x l# ... x n ) is drawn from a single full-covariance 
Gaussian, and model M a , where (x X/ ... x a ) is drawn from two full-covariance 
Gaussians, with (x 1( ... Xi) drawn from the first Gaussian, and (xm, . . . x B ) 
drawn from the second Gaussian. 

Since x x e R d , model Mi has k\ = d + d {d + l) parameters, while model 

M2 has twice as many parameters (k 3 =2k l ) . It can be shown that the i" 1 
frame is a good candidate for a segment boundary if the expression: 



LXUl ^ t — 2 Z 



is negative, where |Z.| is the determinant of the covariance of the whole 
window (i.e.. all n frames), Hfl is the determinant of the covariance of 
the first subdxvision of the window, and |X.| is the determinant of the 
covariance of the second subdivision of the window. 

x / v x ) are established during 

Thus, two subsamples, lx>, ... X.) and (x,.„ ... JC) . are 

step 310 from the window o£ consecutive data samples (x„ . . . x.) . As 
discussed below in a section entitled Improving Efficiency of BIC Tests, a 
number of tests are performed during steps 315 through 328 to 
some BIC tests in the window, when they correspond to locations where the 
detection of a boundary is very unlixely. Specifically, the value of, 
variable a is initialised during step 335 to a value of n/r-1, where r 
the detection resolution <in frames). Thereafter, a test is performed 
during step 320 to determine if the value a exceeds a maximum value. 0_. 

• i- •*■?(■> rhat the value a exceeds a maximum 
If it is determined during step 3 20 tnat cne 

value. «_. then the counter i is set to . value f <«-«_♦ I. r durrng 
step 324. If, however, it is determined during step 320 that the value a 
does not exceed a maximum value, then the counter i is set to a value 

of r during step 328. Thereafter, the difference in BIC values rs 
calculated during step 330 using the equation set forth above. 

A test is performed during step 340 to determine if the value of i equals 
„.r. in other words, have .11 possible samples in the window been 
evaluated. If it is determined during step 340 that the value of i do. 
eva , „ „ f : is incremented by r during step 350 

not yet equal n-r, then the value of i is rncremente y 

to continue processing for the next sample in the window at step 3 If. 
however, it rs determined during step 340 that the value of i equ n-r, 
Chen a further test is performed during step 360 to determrne if 
smallest drfference in BIC values (ABIC. is negative. « it : » 
determined during step 360 that the smallest drfference rn B C val is 
not negative, then the window size is increased durrng step 3 5 before 
returning to step 310 to oonsrder a new window in the manner described 
ebove. Thus, the window site, n, is only increased when the ABIC values 

• a been computed and none of them leads to a 

for all i in one window have been compute 

negative ABIC value. 



If, however, it is determined during seep 360 that the smallest difference 
in BIC values is negative, then i 0 is selected as a segment boundary during 
step 370. Thereafter, the beginning of the new window is moved to i 0 + l and 
the window size is set to N 0 during step 375, before program control 
returns to step 310 to consider the new window in the manner described 
above . 

Thus, the BIC difference test is applied for all possible values of i, and 

i 0 is selected with the most negative ABICi. A segment boundary can be 

detected in the window at frame i: if ABIC i0 < 0, then x i0 corresponds to a 
segment boundary. If the test fails then more data samples are added to 

the current window (by increasing the parameter n) during step 3 60, in a 
manner described below, and the process is repeated with this new window of 
data samples until all the feature vectors have been segmented. Generally, 
the window size is extended by a number of feature vectors, which itself 
increases from one window extension to another. However, a window is never 
extended by a number of feature vectors larger than some maximum value. 
When a segment boundary is found during step 3 70, the window extension 
value retrieves its minimal value (N 0 ) . 

The segmentation subroutine 3 00 is followed by the clustering subroutine 
400. Thus, missing segments is a more severe error than introducing 
spurious segments since clustering can take care of eliminating spurious 
segment boundaries from the segmentation subroutine 300. Indeed, even 
without clustering, in applications like speaker identification, spurious 
boundaries (assuming no speaker identification errors) lead to consecutive 
segments being labeled the same, which is tolerable. Missed boundaries, on 
the other hand, leads to two problems. First, one of the speakers cannot be 
identified. In addition, the other speaker will also be poorly identified 
since that speaker's audio data is corrupted by data from the missed 
speaker . 

A new window selection scheme is presented that improves the overall 
accuracy, especially on small segments. The choice of the window size on 
which the segmentation subroutine 300 is performed is very important. If 
the selected window contains too many vectors, s:me boundaries are likely 
to be missed. If, on the other hand, the selected window is too small, 
lack of information will result in poor representation of the data by the 
Gauss ians . 

It has been suggested to add a fixed amount of data to the current window 
if no segment boundary has been found. Such a scheme does not take 
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ndow Size more onerously when boundaries are not very h . » « 
cne „ ^ a crn -,ii size is considered 

x a window of vectors of a small size 

current window, the sis. of the window i, increased by AH, frames. If no 
ZlZly is found in this new window, the number of fram.s is rnorea.ed by 

s 1- Ai - 9 /5 until a segment boundary is 
• ^ " h «; a : ta'ched 1 Li-urn sis. Un order to avoid 

found or the window extensron ha e. bed a in „«„ ina window 

accuracy problem if a boundary occurs) . This en. ,.„„ 
2 is pretty alow when the window is still small, and rs faster 

111 Z IZZ sets bi 99 er. When . secant boundary is found in a wrndow. 
HI „.*; wrndow begins after the detected boundary. u.in 9 the mr„r„al 
window size (N©) . 

„e„ts in the overall process! , u- are obtained, better^ ^ ^ 

. ir::: :: a™., .xi— <™r 

"Prions where the detection of a ^ » ™ £ 
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small gain is repeated over segment detections and actually 
negligible performance impact). 

„„■ • » when the current window is large, if all the BIC tests are 
in addition, when the curre - he window will have 

performed, the BIC computations at the beginning the 

„ , B «1 times w<th some new information added each time 

been done several times, w-<- example, in a 

has been found in the first 5 seconds, for exampi 
segment ooundary has been to unlike ly that a boundary will be 

window size of 10 seconds, it is quite unlikely tn 

• ^ first 5 seconds with an extension of the .urrenc 
hypothesized in the first b seco _, ons can b e decreased by 

second window. Thus, the number of BIC ^^^//j^;^ 
snoring BIC computations in the beginning ofthe current „i ^ 
. window extension,. In fact, the maxima* number of BIC < *J ^ 
now an adjustable parameter, tweaked according to the speed/ 

recfuired (CU** in FIG. 3) - 

even if no segment boundary has been found yet, 
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big enough one knows that there is no segment present in the first frames. 
This information can be used to do other processing on this part of the 
speech signal. 

The BIC formula utilizes a penalty weight parameter, X, in order to 
compensate for the differences between the theory and the practical 

application of the criterion. It has been found that the best value of X 
that gives a good tradeoff between miss rate and false-alarm rate is 1.3. 
For a more comprehensive study of the effect of X on the segmentation 
accuracy for the transcription of broadcast news, see, A. Tritschler, "A 
Segmentation-Enabled- Speech Recognition Application Using the BIC," M.S. 
Thesis, Institut Eurecom (France, 1998) . 

While in principle the factor X is task-dependent and has to be retuned for 
every new task, in practice the algorithm has been applied to different 
types of data and there is no appreciable change in performance by using 

the same value of X . 

The clustering subroutine 400 attempts to merge one of a set of clusters 
Ci, C K , with another cluster, leading to a new set of clusters C x ' , 

C R -i ' , where one of the new clusters is the merge between two previous 
clusters. To determine whether to merge two clusters, C A and Cj , two 

models are built: the first model, M x , is a Gaussian model computed with 
the data of C t and C } merged which leads to BICi. The second model, M 2 , 
keeps two different Gauss ians, one for Ci and one for C if and gives BIC 2 . 
Thus, it is better to keep two distinct models if ABIC = BIC X - BIC 2 < 0. 
If this difference of BIC is positive, the two clusters are merged and we 
have the desired new set of clusters . 

S. Chen and P. Gopalakrishnan, "Speaker, Environment and Channel Change 
Detection and Clustering Via the Bayesian Information Criterion, M 
Proceedings of the DARPA Workshop, 1998, shows how to implement off- line 
clustering in a bottom-up fashion, i.e., starting with all the initial 
segments, and building a tree of clusters by merging the closest nodes of 
the tree (the measure of similarity being the BIC) . The clustering 
subroutine 400 implements a new online technique. 

As discussed below in conjunction with FIG. 4, the online clustering of the 

present invention involves first the K clusters found in the previous 

iterations (or calls to the clustering procedure 400), and the new M 
segments to cluster. 



v«r- classification process 200 implements 
As previously indicated, the speaker classic P 

or a cluster identified during a previous iteration 
subroutine 400. 

h inPI c 4 Che clustering subroutine 400 initially collects the M 
As shown in FiG. 4. ^ ^ 

- — <° - ClUSter t d The uLrZ subroutine 400 calculates the 
Por ell unentered segment , t h e us 

difference in BIC values relative to all 
segments during step 420. as follows, 

«-ho clustering subroutine 400 
In addition, for all unentered segments, the . 

also calculates the difference in BIC values relative to the K 
clusters during step 430, as follows: 

+ 1^ + iSifii) log. 

T „ere,f=er, the clustering subroutine 400 identifies the ^^«« 

, s ABIC among the M(MeK-l) results during step 440. A 
in BIC values. ABIC am g determine if the largest 

" the " Pe££ °™r Ib^C is Positive. » discussed further below. 
Terence in BIC ve lues - ^ _ _ ^ pQssible 

the ABIC„* value is the xargesc clus tered. It is 

— r rrrr;^ ::: ~ - segment. ^ — 

„ oc only ^ ™ attempt to merge the segment with a cluster 

X luster, but rather the clustering subroutine 400 

i:;::.! a„ ^ ^ «- - 

„ lt is aetermined during step 450 that the largest difference in BIC 

, ABIC is positive, then the current segment is merged with the 

values, ABIC^c, is p segment is 

listing cluster and the value of M is incremented, or .he 
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merged with another unclustered segment and the value of K is incremented, 

and the value of M is decremented by two, during step 460. Thus, the 
counters are updated based on whether there are two segments and a new 

cluster has to be created (M=M-2 and K=K+1), because the two segments 
correspond to the same cluster, or if one of the entities is already a 

cluster, then the new segment is merged into the cluster (M=M- 1 and K is 
constant). Thereafter, program control proceeds to step 480, discussed 
below . 

If, however, it is determined during step 450 that the largest difference 
in BIC values, ABICa«, is not positive, then the current segment is 
identified as a new cluster, and either (i) the value of the cluster 
counter, K, is incremented and the value of the segment counter, M, is 
decremented, or (ii) the value of the cluster counter, K, is incremented by 
two and the value of the segment counter, M, is decremented by two, during 
step 470, based on the nature of the constituents of ABIC^ . Thus, the 
counters are updated based on whether there is one segment and one existing 
cluster (M=M-1 and K=K+1) or two new segments {M=M-2 and K=K+2) . 

Thereafter, a test is performed during step 480 to determine if the value 
of the segment counter, M, is strictly positive, indicating that 
additional segments remain to be processed. If it is determined during 
step 480 that the value of the segment counter, M, is positive, then 
program control returns to step 440 to continue processing the additional 
segment is) in the manner described above. If, however, it is determined 

during step 480 that the value of the segment counter, M, is zero, then 
program control terminates . 

The clustering subroutine 400 is a suboptimal algorithm in comparison to 
the offline bottom-up clustering technique discussed above, since the 

maxima considered for the ABIC values can be local in the offline scheme, 
as opposed to the global maxima found in the online version. Since the 
optimal segment merges are usually those corresponding to segments close in 
time, the online clustering subroutine 400 makes it easier to associate 
such segments to the same cluster. In order to reduce the influence of the 
non-reliable small segments to cluster, only the segments with sufficient 
data are clustered; the other segments are gathered in a separate 'garbage* 
cluster. Indeed, the small segments can lead to errors in clustering, due 
to the fact that the Gaussians could be poorly represented. Therefore, in 
order to improve the classification accuracy, the small segments are all 
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CLAIMS 



1. A method for tracking a speaker in an audio source, said method 
comprising the steps of: 

identifying potential segment boundaries in said audio source; and 
clustering homogeneous segments from said audio source substantially 
concurrently with said identifying step. 

2. A method as claimed in claim 1, wherein said identifying step 
identifies segment boundaries using a BIC model-selection criterion. 

3. A method as claimed in claim 1 or claim 2, wherein a first model 
assumes there is no boundary in a portion of said audio source and a second 
model assumes there is a boundary in said portion of said audio source. 

4. A method as claimed in any of claims 1 to 3 , wherein a given sample, 
i, in said audio source is likely to be a segment boundary if the following 
expression is negative: 

ABIQ = - f lo$L w \ + \ loglZ/l + log|2j 

, 1 j|0 ^ did ± 1Y ] . 
+ 2^\d + 2 J lo S n 

where l£ w l is the determinant of the covariance of the window of all n 

samples, |Z t *l is the determinant of the covariance of the first subdivision 

of the window, and |Z S | is the determinant of the covariance of the second 
subdivision of the window. 

5. A method as claimed in any preceding claim, wherein said identifying 

step considers a smaller window size, n, of samples in areas where a 
segment boundary is unlikely to occur. 

6 . A method as claimed in claim 5, wherein said window size, n, is 

increased in a relatively slow manner when the window size is small and 
increases in a faster manner when the window size is larger. 

7. A method as claimed in claim 5, wherein said window size, n, is 
initialized to a minimum value after a segment boundary is detected. 

8. A method as claimed in claim 2, wherein said BIC model selection test 
is not performed at the border of each window of samples. 
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* m ethod as claimed in claim 2, wherein said BIC model selection test 
c ? ::Lr,ed when the window size, n, exceeds a predefined threshold. 

tu a method as claimed in any preceding claim, wherein said clustering 
step is performed using a BIC model-selection criterion. 

L1 a method as claimed in claim 10, wherein a first model assumes that 
£0 s gments or clusters should he merged, and a second mode assumes that 
said »o segments or clusters should be maintained independently. 

12 a method as claimed in claim 11, further comprising the step of 
ZrJ 9 lard two clusters if a difference in BXC values for each of sard 
models is positive. 

U » „. th =a as claimed in any pr.=«din 3 clai., wherein said c lu s«rin s 
st ep 1. per £ o».d usins K pr.viouslv l^n.ifi- clears and M s.p»e„c, „ 
be clustered. 

14 A method as claimed in any preceding claim, further comprising the 
steP of assigning a cluster identifier to each of said clusters. 

15 a method as claimed in any preceding claim, further comprising the 

15. a mecnu k identif icatxon engine 
step of orocessing said audio source with a speaker 

to assign a speaker name to each of said clusters. 

16 . A method for tracking a speaker in an audio source, said method 
comprising the steps of: 

ratifying potential segment boundaries in said audio source; and 

catering segments from said audio source corresponding to the same 
speaker substantially concurrently with said identifying step. 

17 A method as claimed in claim 16. wherein said identifying step 
Identifies segment boundaries using a BXC model -selection criterion. 

n • n uh prein a first model assumes there 
18. A method as claimed in claim 17, wherein a 
i. no boundary in a portion of said audio source and a second 
there is a boundary in said portion of said audio source. 

19 a method as claimed in claim 16, wherein said identifying step 
Considers a smaller window size, n. of samples in areas where a segment 
boundary is unlikely to occur. 



20. A method as claimed in claim 17, wherein said BIC model selection 
test is not performed where the detection of a boundary is unlikely to 
occur. 

21. A method as claimed in claim 16, wherein said clustering step is 
performed using a BIC model-selection criterion, where a first model 
assumes that two segments or clusters should be merged, and a second model 
assumes that said two segments or clusters should be maintained 
independent ly . 

22. A method as claimed in claim 16, wherein said clustering step is 

performed using K previously identified clusters and M segments to be 
clustered . 

23. A method for tracking a speaker in an audio source, said method 
comprising the steps of: 

identifying potential segment boundaries during a pass through said 
audio source; and 

clustering segments from said audio source corresponding to the same 
speaker during said same pass through said audio source. 

24. A method as claimed in claim 23, wherein said identifying step 
identifies segment boundaries using a BIC model-selection criterion. 

25. A method as claimed in claim 24, wherein a first model assumes there 
is no boundary in a portion of said audio source and a second model assumes 
there is a boundary in said portion of said audio source. 

26. A method as claimed in claim 23, wherein said identifying step 
considers a smaller window size, n, of samples in areas where a segment 
boundary is unlikely to occur. 

27. A method as claimed in claim 24, wherein said BIC model selection 
test is not performed where the detection of a boundary is unlikely to 
occur . 

28. A method as claimed in claim 23, wherein said clustering step is 
performed using a BIC model-selection criterion, where a first model 
assumes that two segments or clusters should be merged, and a second model 
assumes that said two segments or clusters should be maintained 
independently. 
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29 a method as claimed in claim 23, wherein said clustering step is 
performed using K previously identified clusters and M segments to be 
clustered . 

30. A system for tracking a speaker in an audio source, comprising: 
a memory that stores computer-readable code; and 

a processor operatively coupled to said memory, said processor 
configured to implement said computer-readable code, said co^uc.r-r^bl. 
code configured to: identify potential segment boundaries in said audio 
source ; and 

cluster homogeneous segments from said audxo source substantially 
concurrently with said identification of segment boundaries. 

31. An article of manufacture, comprising: 

a computer readable medium having computer readable code means 
ambcdied thereon, said computer readable program code means comprising: 

a step to identify potential segment boundaries in said audio source; 



and 



a step to cluster homogeneous segments from said audio source 
substantially concurrently with said identification of segment boundaries. 

32. A system for tracking a speaker in an aud.o source, comprising: 

3Q a memory that stores computer-readable code; and 

a processor operatively coupled to said memory, said processor 
. confined to implement said computer -readable code, said computer- readable 
code configured to: 

identify potential s.g»=nt boundaries in said audio souroe; and 

du.wr slants fro. said audio souro. corresponds to t>. sa». 
speaker substantially oonourr.ntiy with .aid identification of se,»ent 
40 boundaries. 

33 . A computer program comprising program code to ^ ^ into a 
computer and executed, cause the computer to perform the steps 
according to any one of claims 1 to 30. 



